1. U.S. Pat. No. 5,315,700 entitled, xe2x80x9cMethod and Apparatus for Rapidly Processing Data Sequencesxe2x80x9d, by Johnston et. al., May 24, 1994
2. U.S. Pat. No. 6,130,967 entitled, xe2x80x9cMethod and Apparatus for a Reduced Instruction Set Architecture for Multidimensional Image Processingxe2x80x9d, by Shih-Jong J. Lee, et. al., Oct. 10, 2000
3. Pending application Ser. No. 08/888,116 entitled, xe2x80x9cMethod and Apparatus for Semiconductor Wafer and LCD Inspection Using Multidimensional Image Decomposition and Synthesisxe2x80x9d, by Shih-Jong J. Lee, et. al., filed Jul. 3, 1997
4. U.S. Pat. No. 6,122,397 entitled, xe2x80x9cMethod and Apparatus for Maskless Semiconductor and Liquid Crystal Display Inspectionxe2x80x9d, by Shih-Jong J. Lee, et. al., Sep. 19, 2000
5. U.S. Pat. No. 6,148,099 entitled, xe2x80x9cMethod and Apparatus for Incremental Concurrent Learning in Automatic Semiconductor Wafer and Liquid Crystal Display Defect Classificationxe2x80x9d, by Shih-Jong J. Lee et. al., Nov. 14, 2000
1. U.S. patent application Ser. No. 09/693,723, xe2x80x9cImage Processing System with Enhanced Processing and Memory Managementxe2x80x9d, by Shih-Jong J. Lee et. al, filed Oct. 20, 2000
2. U.S. patent application Ser. No. 09/693,378, xe2x80x9cImage Processing Apparatus Using a Cascade of Poly-Point Operationsxe2x80x9d, by Shih-Jong J. Lee, filed Oct. 20, 2000
3. U.S. patent application Ser. No. 09/692,948, xe2x80x9cHigh Speed Image Processing Apparatus Using a Cascade of Elongated Filters Programmed in a Computerxe2x80x9d, by Shih-Jong J. Lee et. al., filed Oct. 20, 2000
4. U.S. patent application Ser. No. 09/703,018, xe2x80x9cAutomatic Referencing for Computer Vision Applicationsxe2x80x9d, by Shih-Jong J. Lee et. al, filed Oct. 31, 2000
5. U.S. patent application Ser. No. 09/702,629, xe2x80x9cRun-Length Based Image Processing Programmed in a Computerxe2x80x9d, by Shih-Jong J. Lee, filed Oct. 31, 2000
6. U.S. Patent Application entitled, xe2x80x9cStructure-guided Image Measurement Methodxe2x80x9d by Shih-Jong J. Lee et. al., filed Dec. 15, 2000.
1. Lee, J S J, Haralick, R M and Shapiro, L G, xe2x80x9cMorphologic Edge Detection,xe2x80x9d IEEE Journal of Robotics and Automation RA-3 No.2:142-56, April, 1987.
2. Haralick R M and Shapiro, L G, xe2x80x9cSurvey Image Segmentation Techniques,xe2x80x9d Comput. Vision, Graphics, and Image Processing, vol. 29 No. 1: 100-132, January 1985.
3. Otsu N, xe2x80x9cA Threshold Selection Method from Gray-level Histograms,xe2x80x9d IEEE Trans. System Man and Cybernetics, vol. SMC-9, No. 1, January 1979, PP 62-66.
4. Serra, J, xe2x80x9cImage Analysis and Mathematical Morphology,xe2x80x9d London: Academic Press, pp 319-321, 1982.
5. Sternberg, S R, xe2x80x9cGrayscale Morphology,xe2x80x9d Comput. Vision, Graphics, and Image Processing, vol. 35 No. 3: 333-355, September 1986.
This invention relates to image processing methods that incorporate knowledge of object structure derived from the image itself or from a-priori knowledge of an object""s structural relationships from its design data (such as CAD drawings) to enhance object features and/or guide image measurement estimation and object detection.
Common tasks in computer vision applications include enhancement and detection of objects of interest, refinement of detected object masks, and measurement, alignment or classification of the refined object. Other applications include enhancement for image compression or image highlighting for display. Many computer vision applications require the enhancement and measurement of image features for objects of interest characterization or detection. Application domain knowledge is available in most computer vision applications. The application domain knowledge can often be expressed as structures of image features such as shaped color, edges, lines and regions, or changes with time such as object motion on a prescribed path. The structures include spatial relationships of object features such as shape, size, intensity distribution, parallelism, co-linearity, adjacency, position, etc. The structure information can be particularly well defined in industrial applications such as semiconductor, electronic or machine part inspections. In machine part inspections, most of the work-pieces have available Computer Aided Design (CAD) data that specifies CAD components as entities (e.g. LINE, POINT, 3DFACE, 3DPOLYINE, 3DVERTEX, LINE, POINT, 3DFACE, 3DPOLYLINE, 3DVERTEX, etc.) and blocks (properties that are associated) of entities. Semiconductor applications frequently have step and repeat type processes that form lines, patterns, and mosaic structures. In biomedical or scientific applications, structure information may also be loosely defined. For example, a cell nucleus is generally round, frequently stains dark, and different but known approximate shapes can differentiate different types of blood cells or chromosomes.
The capability of a computer vision system is often characterized by its detection/measurement accuracy, repeatability and throughput. It is desirable to achieve sub-pixel measurement accuracy and repeatability for many computer vision applications. Application domain knowledge used according to this invention can significantly improve the capability of a computer vision system to make accurate and repeatable measurements. However, it is non-trivial to efficiently use the application domain knowledge in high precision applications.
Prior art uses an image segmentation approach for image feature detection or measurement (Haralick R M and Shapiro, L G, xe2x80x9cSurvey Image Segmentation Techniquesxe2x80x9d, Comput. Vision, Graphics, and Image Processing, vol. 29 No. 1: 100-132, January 1985). The image segmentation approach converts a grayscale image into a binary image that contains object of interest masks. Binary thresholding is a common technique used in the image segmentation approach to create masks.
Because edges or features of an image are imaged by the optical and imaging system as continuously varying gray levels, there exists no single gray level that represents edge pixels. For this reason, any system that depends on taking a binary threshold of the image before critical dimensions are determined must necessarily introduce quantization errors into the measurement. Binary thresholding also exacerbates the resolution limiting effect of system noise.
Prior art applies application domain structure information through a projection/dispersion approach. The projection/dispersion approach integrates image pixel values in a pre-defined direction in the image. This can be done using a binary image (projection) or grayscale image (dispersion) and results in a one-dimensional plot of summed pixel values. The application domain structure information defines the projection directions, however misalignments, variations in illumination, and image noise limit the resolving capability of these projections. The prior art approach is sensitive to system variations such as rotation, object illumination, changes in object surface texture (which affects gray levels), etc. Rotation errors result in the integration of pixel values along a wrong direction that is destructive to accuracy. Furthermore, the projection-based approach cannot effectively combine multiple two-dimensional structure information (such as anti-parallelism, orthogonality, intersection, curvaceous qualities) where features of interest may be along different directions or complex. Another difficulty in the prior art is that two-dimensional processing is needed for reliable sub-pixel accuracy due to the utility of using as many pixels as possible for the measurement. Use of all possible pixels minimizes spatial quantization errors and also aids reconstruction and interpolation between sample values. Herein there are two difficulties, the prior art does not take advantage of all pixels whose position is related, the prior art confuses image surface information and image edge information through the use of projection, and the projections cannot be used effectively with complex structures. Where the prior art could have employed two dimensions to achieve a better result (but not a projection result), such grayscale processing is in the prior art computationally demanding and requires expensive and extensive special hardware to achieve desired throughput. Additionally, in an effort to enhance image features and thereby improve measurement signal to noise or object detection or classification accuracy, prior art uses linear filters. Linear filters are derived from a digital signal processing paradigm where structure information is considerably less obvious. Linear filters are not designed to input structure information and therefore cannot utilize application domain structure knowledge. Where linear filters have been used in the prior art for feature enhancement, their own characteristics obscure essential image characteristics because they introduce phase delay distortion that causes image blur, under-shoot, over-shoot or ringing and edge displacement. These image distortions increase uncertainty of feature measurement. Image variability and noise in conjunction with prior art linear filtering and thresholding seriously degrade measurement reliability and accuracy.
It is an object of this invention to provide improved image feature extraction, and feature enhancement through a structure-guided image processing method. It is another object of the invention to enhance image features through use of nonlinear image processing that does not introduce phase shift and/or blurry effect (transient aberration). A further object of the invention is to provide methods for utilizing application domain knowledge encoded into the image processing parameters for structure-guided extraction and enhancement of features of interest and/or to remove noisy or irrelevant information. It is a further object of the invention to create an object mask image output from the structure guided image feature enhanced image. A further object is to provide a weight image output derived from the structure guided image feature enhanced image. The weight image output and/or the mask image outputs can be used for image compression, highlighting and display of an image, measurement of objects within the feature enhanced image, or object detection.
Structure guided morphological processing uses a-priori geometric structure information to tailor or optimize the structuring elements used to extract image features of particular interest to the module. Structure-guided morphological processing involves a variety of morphological operations with various size and shaped structuring elements, that, once applied to the image, highlight specific shape or position features that are of particular importance to the function of the algorithm. This invention seeks to provide high performance image feature extraction and enhancement through a structure-guided image processing method.